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Section 2 



Random variables and their 
properties. Expectation. 

Let {il,A,F) be a probability space and {S,B) be a measurable space where S is a cr-algebra of subsets of 
<S. A random variable X : Q — > <S is a measurable function, i.e. 

B e X-^{B) e A. 

When S — R we will usually consider a a-algcbra B of Borel measurable sets generated by sets Uj<n('^»' 
(or, equivalently, generated by sets {ai,bi) or by open sets). 

Lemma 3 X : Q is a random variable iff for allt&M. 

{X < t} := {iv G fl : X{u)) G {-oo,t]} G A. 

Proof. Only <^= direction requires proof. We will prove that 

v = {Dcm: x-\D) e A} 

is a cr-algebra. Since sets {—oo,t] G V this will imply that B CT>. The result follows simply because taking 
pre-image preserves set operations. For example, if we consider a sequence Dj e X> for i > 1 then 

^^'(U^O = \JX-\Di)GA 

i>l i>l 

because X~^{Di) e A and >l is a a-algebra. Therefore, Ui>i^j ^ Other properties can be checked 
similarly, so 2? is a cr-algebra. 

□ 

Let us define a measure Fx on B by ¥x =Vo X~^, i.e. for B G B, 

Vx{B) = P(X e B) = F{X-\B)) = P o X-\B). 

{S,B,Vx) is called the sample space of a random variable X and Px is called the law of X. Clearly, on this 
space a random variable : S ^ S defined by the identity ^(s) = s has the same law as X. 

When 5 = M, a function F{t) = P{X < t) is called the cumulative distribution function (c.d.f.) of X. 

Lemma 4 F is a c.d.f. of some r.v. X iff 

1. 0< F{t) < 1, 

2. F is non- decreasing, right- continuous, 
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3. limt__oo F{t) = 0, limt^+oo F{t) = 1. 



Proof. The fact that any c.d.f. satisfies properties 1 - 3 is obvious. Let us show that F which satisfies 
properties 1 - 3 is a c.d.f. of some r.v. X. Consider algebra A consisting of sets Ui<n('^i'^«] ^'^'^ disjoint 
intervals and for all n > 1 . Let us define a function P on A by 

One can show that P is countably additive on A. Then, by Caratheodory extension Theorem 1, P extends 
uniquely to a measure P on a{A) — B - Borel measurable sets. This means that (M,K,P) is a probability 
space and, clearly, random variable X : M ^ M defined by X{x) = x has c.d.f. V{X < t) — F{t). Below we 
will sometimes abuse the notations and let F denote both c.d.f. and probability measure P. 
Alternative proof. Consider a probability space ([0, 1],B, A), where A is the Lebesgue measure. Define r.v. 
X : [0, 1] ^ M by the quantile transformation 

X{t) = inf{x e M, F{x) > t}. 

The c.d.f. of X is X{t ■.X{t)<a)^ F{a) since 

X{t) < a inf{x : F{x) > t} < a 3a„ a, F(a„) > t F{a) > t. 



Figure 2.1: A random variable defined by quantile transformation. 



Definition. Given a probability space (fi. A, P) and a r.v. X : Q. ^ S\ei (j{X) be a cr-algebra generated 
by a collection of sets {X^^{B) : B E B}. Clearly, a{X) C A. Moreover, the above collection of sets is itself 
a cr-algebra. Indeed, consider a sequence Ai = X^^{Bi) for some Bi E B. Then 

U A, = U X-\B,) = X-U\J b) = X-\B) 



where B G lJi>i ^ ^- '^{-^) called the a-algehra generated by a r.v. X. 



1/2 1 

Figure 2.2: (7(A) generated by X. 
Example. Consider a r.v. defined in figure 2.2. We have P(A = 0) = 5,P(A = 1) = 5 and 

a(A) = {0, [0, ^],(^,l]jO,l]}. 
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Lemma 5 Consider a probability space (fi, .4, P), a measurable space (<S, B) and random variables X : — > <S 

and y : f2 ^ M. Then the following are equivalent: 

1. Y = g{X) for some (Borel) measurable function g : <S ^ M. 

2. F : f2 — > R is measurable on {Q.,(j{X)), i.e. with respect to the a-algebra generated by X. 

Remark. It should be obvious from the proof that M can be replaced by any separable metric space. 

Proof. The fact that 1 implies 2 is obvious since for any Borel set -B C K the set B' := g~^{B) G B 
and, therefore, 

{Y = g{X) eB} = {Xe g-\B) = B'} = X-\B') e a{X). 
Let us show that 2 implies 1. For all integer n and k consider sets 



By 2, An,k e a{X) = : B e B} and, therefore, A„,fc = for some B„^h € B. Let us 

consider a function 

9n{X) = Y,^l{X€Br,,k). 



By construction, \Y — gn{X)\ < ^ since 



It is easy to see that gn{x) < gn+i{x) and, therefore, g{x) = lim„^oo fi'n(a;) is a measurable function on 
{S,B) and, clearly, Y = g{X). 

□ 

Discrete random Vciriables. 

A r.v. X : fl ^ S is called discrete if ^x{{Si}i>i) = 1 for some sequence Si G S. 

a 

Absolutely continuous random variables. 

On a measure space {S,B), a measure P is called absolutely continuous w.r.t. a measure A if 

\/B G B, X{B) = ^ ¥{B) = 0. 
The following is a well known result from measure theory. 

Theorem 2 ( Radon- Nikodym) Iff and A are sigma-finite and P is absolutely continuous w.r.t. A then there 
exists a Radon-Nikodym derivative / > such that for all B £ B 

P(B) = / /(s)dA(s). 
Jb 

f is uniquely defined up to a X-null sets. 

In a typical setting of <S = IR'^, a probability measure P and Lebesgue's measure A, / is called the density of 
the distribution P. 

□ 

Independence. 

Consider a probability space {Q,C,F) and two u-algebras A,B CC. A and B are called independent if 

¥{AnB)=¥{A)¥{B) for all AeA,B€B. 
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(T-algebras Ai C C for i < n are independent if 

P(^i n • • • n An) = Yl H^t) for all A^ e Ai. 

cr-algebras Ai C C for i < n are pairwise independent if 

F{Ai n Aj) = ¥{A,)P{Aj) for all A, e A^, A^- e Aj, i 7^ 

Random variables Xi : CI ^ S for i < n are (pairwise) independent if u-algebras a{Xi)^ i < n are (pairwise) 
independent which is just another convenient way to state the familiar 

P(Xi e Bi, . . . ,X„ e B„) = P(Xi e Bi) X . . . X P(X„ e B„) 

for any events _Bi , . . . , i3„ G B. 

Example. Consider a regular tetrahedron die, Figure 2.3, with red, green and blue sides and a red-green- 
blue base. If we roll this die then indicators of different colors provide an example of pairwise independent 
r.v.s that are not independent since 

P(r) = P(fe) = V{g) = ^ and V{rb) = P(rg) = V{bg) ^ ^ 

but 

P(r6g)-^^P(r)P(6)P(5)= (^)'. 









b\ 











Figure 2.3: Pairwise independent but not independent r.v.s. 

□ 

Independence of cr-algebras can be checked on generating algebras: 
Lemma 6 If algebras Ai,i < n are independent then a-algebras a{Ai) are independent. 
Proof. Obvious by Approximation Lemma 2. 

□ 

Lemma 7 Consider r.v.s Xi : R on a probability space (fl,A,P). 

1. Xi's are independent iff 

V{Xi <ti,...,Xn<tn)^ V{Xi < ti) X . . . X P(X„ < i„). (2.0.1) 

2. If the laws of Xi 's have densities fi{x) then Xi 's are independent iff a joint density exists and 

f{xi, ...,Xn) = YifiiXt). 
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Proof. 1 is obvious by Lemma 6 because (2.0.1) implies the same equality for intervals 

P(Xi e (ai, . . . , X„ e (a„, bn]) = P(Xi e (ai, 6i]) x . . . x P(X„ e {an, 

and, therefore, for finite union of disjoint such intervals. To check this for intervals (for example, for n = 2) 
we can write P(ai < Xi < bi,a2 < < 62) as 

P(Xi < bi,X2 < 62) -P(^i < ai,X2 < 62) -P(Xi < bi,X2 < as) +P(Xi < ai,X2 < 02) 
= P(Xi < bi)F{X2 < &2) - P(^i < ai)¥{X2 < 62) - P(Xi < bi)¥{X2 < 02) + P(Xi < ai)P(X2 < 02) 
= (P(Xi < 61) - P(Xi < ai)) (P(X2 < 62) - P(^2 < ^2)) = P(oi < ^1 < 6i)P(a2 < ^2 < ?>2)- 

To prove 2 we start with " <^=" . 

P(n{x, G ^J) = P(x e ^1 X • • • X A„) = / Yl Mxi)dx 

= TT / fi{xi)dxi {by Fubini's Theorem} = TT ¥{Xi e Ai). 
Next, we prove "=>". First of all, by independence, 

P(x e X • • • X A„) = \[nXi e ^i) '^="' / n fi^^i)'^''- 

Therefore, the same equality holds for sets in algebra A that consists of finite unions of disjoint sets Ai x 
• • • X Aji, i.e. 

P(X gB)= I W fi{xi)dx for BgA. 
Both V{X G -B), /b n fi{xi)dx are countably additive on A and finite, 

P(M")= / T\fi{xi)dx = l. 

By the Caratheodory extension Theorem 1, they extend uniquely to all Borel sets B = <j(A), so 

P(B)= / \\fi{xi)dxior BeB. 

JB 

□ 

Expectation. If X : ^ M is a random variable on (n,^,P) then expectation of X is defined as 

EX^ [ X{u))dF{w). 

In other words, expectation is just another term for the integral with respect to a probability measure and, 
as a result, expectation has all the usual properties of the integrals. Let us emphasize some of them. 

Lemma 8 1. If F is the c.d.f. of X then for any measurable function g : M ^ M, 

Eg{x) = / 9ix)dF{x). 

Jr 

2. If X is discrete, i.e. F{X e {xi}i>i) = 1, then 

EX = Y,Xi¥{X = Xi). 

i>\ 
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3. IfX-.fl^R'' has a density f{x) on andg-.R'' then 

EgiX) = J g{x)f{x)dx. 

Proof. All these properties follow by making a change of variables x = X{oj) or a; = X~^{x), i.e. 

E5(X)= / ff(XH)(iPH= / g{x)dVoX-\x)= [ g{x)dFx{x), 
Jq Jm Jm 

where ¥x = P o X^^ is the law of X. Another way to see this would be to start with indicator functions of 
sets g{x) = l{x £ B) for which 

Ec;(X) = P(X e S) = Px(S) = / I(x e B)dFx{x) 

Jm 

and, therefore, the same is true for simple step functions 

9i^) = ^WiKx e Bi) 

i>n 

for disjoint Bj. By approximation, this is true for any measurable functions. 

□ 
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